On most computing devices, the user provides inputs with a keyboard and with a conventional pointing device, such as a mouse, trackball, scratchpad, or other pointing button. Such input devices are much less intuitive than touch screens that enable a user to touch a spot on a displayed image and provide an input related to the object or move the virtual object that is being touched, to different positions on the display. However, touch screens are somewhat limited in the scope of user input and interaction that they can provide. For example, most touch screens are only able to detect a single point that is being touched on the screen at a time and are only responsive to a touch on the screen, which precludes more complex input and interaction with the computing device.
A further evolution of techniques to facilitate human-machine interaction exploits computer vision technology. For example, the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST October 1997:14-17,” has developed another form of “keyboardless” machine interface. The metaDESK includes a generally planar graphical surface that not only displays computing system output, but also receives user input by recognizing pattern-encoded objects placed against the graphical surface. The graphical surface of the metaDESK is facilitated using infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors disposed beneath the surface of the metaDESK. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The IR camera can detect IR reflections from the undersurface of the objects placed on the graphical surface.
Similarly, papers published by Jun Rekimoto of the Sony Computer Science Laboratory, Inc., and associates describe a “HoloWall” and a “HoloTable” that use IR light to detect objects. It should be noted that the papers describing the “HoloWall” and “HoloTable” generally are silent regarding the details and process used for detecting objects based upon the IR light reflected from the object.
Examples of other methods and systems for detecting and responding to objects placed on or near a graphical surface are used in inventions owned by the assignee of the present application. These inventions are described, for example, in co-pending U.S. patent applications, including Ser. No. 10/813,855, entitled “Template Matching On Interactive Surface;” Ser. No. 10/814,577, entitled “Identification Of Object On Interactive Display Surface By Identifying Coded Pattern;” and application Ser. No. 10/814,761 entitled “Determining Connectedness And Offset Of 3D Objects Relative To An Interactive Surface,” all of which were filed on Mar. 31, 2004.
Although inventions such as the metaDESK and the other inventions previously listed respond to objects on the graphical display surface, it is significant that the metaDESK's response is to the contemporaneous placement and movement of the objects. There is no indication in the published literature that the metaDESK can enable a user to input one or more bits by depressing a movable portion of an object. Clearly, it would be desirable to provide such a capability, to enable applications to respond to input not only from an object being placed on and/or moved about on a graphical display surface, but also as a result of a user interacting with the object. The user should be able to interact with the user input device to change a state of one or more bits related to the object for input to an application running on an interactive display surface.
Using the human-machine interfaces described above, moving one's hands or other physical objects on or near to the graphical display surface may replace keyboards, pointing devices, and other more conventional input devices. But, instead of only responding to a static condition in which an object is always recognized in the same manner, it would be more interesting and offer greater functionality if users are able to change the recognition of an object or a condition associated with the object based upon the user's interaction with the object. The altered condition of the object resulting from the user's interaction with it should be detected using the same approach employed for sensing the object on the surface.